Inter-node communication scheme for node status sharing

ABSTRACT

A gossiping scheme for sharing node status in a cluster of nodes provides a robust mechanism for determining node status within the cluster. Nodes transmit gossip messages to each other nodes, the gossip messages listing other nodes in the cluster that are operational. When a node does not receive a gossip message from a particular node within a predetermined time period, then the node transmits messages to the other nodes indicating that the particular node is down. However, if another node has received a packet from the particular node within the predetermined time period and receives the node down message, then the other node responds with a node alive message.

The present application is a Continuation of U.S. patent applicationSer. No. 12/959,581, filed on Dec. 3, 2010 and claims priority theretounder 35 U.S.C. §120. The disclosure of the above-referenced parent U.S.patent application is incorporated herein by reference.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present U.S. patent application is related to co-pending U.S. patentapplication Ser. No. 12/959,568 entitled “DYNAMIC RATE HEARTBEATING FORINTER-NODE STATUS UPDATING” and Ser. No. 12/959,556 entitled“ENDPOINT-TO-ENDPOINT COMMUNICATIONS STATUS MONITORING” filedcontemporaneously herewith and having at least one common inventor andassigned to the same Assignee, the disclosures of which are incorporatedherein by reference.

BACKGROUND

1. Field of the Invention

The present invention is related to node status monitoring indistributed computing systems, and more specifically to a scheme of nodestatus sharing by gossiping among the nodes.

2. Description of Related Art

In large-scale distributed computer systems, such as those usingdistributed software models to perform tasks, multiple nodes provideindependent execution of sub-tasks. In order to keep such a systemoperational, and further, to provide for proper operation of distributedapplications that use the multiple nodes to perform various tasks, thestatus of nodes is tracked. In particular, in order to assign tasks tonodes, and in order to ensure that a node is available to communicatewith to perform a task, the operational status of the nodes and theirability to communicate with the other nodes must be monitored.

Communications and status monitoring is typically centralized, with amonitoring application providing information about node and interfacestatus. The monitoring application may use distributed agents to performthe monitoring on each node. Heartbeat messages are typically sent fromthe nodes to a centralized manager that maintains a record of the statusof each node.

BRIEF SUMMARY

The invention provides a node status monitoring system and method thatis embodied in a computer-performed method. The method communicates nodestatus information among the nodes by gossiping, eliminating the needfor a central clearinghouse or monitoring facility.

The method determines node operating status among a cluster of nodes ofa computer system by transmitting gossip messages directly between nodepairs in the cluster. The gossip messages include an indication of othernodes in the cluster of nodes that are operational. When a node has notreceived a gossip message from another node for a predetermined periodof time, the, the local status for the other node is set tonon-operational, and a message indicating that the other node is down issent to each of the other nodes in the cluster. If another node hasreceived a status message within the predetermined period of time andreceives the node down message, then the other node sends a node alivemessage to the other nodes.

The foregoing and other objectives, features, and advantages of theinvention will be apparent from the following, more particular,description of the preferred embodiments of the invention, asillustrated in the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

The novel features believed characteristic of the invention are setforth in the appended claims. The invention itself, however, as well asa preferred mode of use, further objectives, and advantages thereof,will best be understood by reference to the following detaileddescription of the invention when read in conjunction with theaccompanying Figures, wherein like reference numerals indicate likecomponents, and:

FIG. 1 is a block diagram of a distributed computer system in whichtechniques according to an embodiment of the present invention arepracticed.

FIG. 2 is a pictorial diagram depicting communication between nodes of acomputer system in accordance with an embodiment of the presentinvention.

FIGS. 3A-3C are pictorial diagrams showing communications in a gossipingscheme in accordance with an embodiment of the present invention.

FIG. 4 is a flowchart of a method in accordance with an embodiment ofthe present invention.

DETAILED DESCRIPTION

The present invention encompasses techniques for monitoringcommunication status in clusters of nodes within distributed computingsystems. In particular, embodiments of the invention provide determiningthe operational status of the nodes in the cluster by sending gossipingmessages between the nodes, containing indications of the status ofother nodes. When a node has not received a gossip message from aparticular node for a predetermined time period, the node sends amessage that the particular node is down. If another node has received agossip message from the particular node in the predetermined timeperiod, that other node sends a message to the other nodes indicatingthat the particular node is alive. Thus, the present invention providesa self-correcting and robust node status information sharing system.Other features that may be incorporated in the gossiping methodologywill be described in further detail below.

Referring now to FIG. 1, a distributed computer system in accordancewith an embodiment of the present invention is shown. A first physicalprocessing node 10A includes a processor core 12 coupled to a memory 14that stores program instructions for execution by processor 12. Theprogram instructions include program instructions forming computerprogram products in accordance with embodiments of the invention thatprovide node status information for either physical processing nodes10A-10D, virtual processing nodes partitioned within the depictedcomputer system as will be described in further detail below, or both.Processing node 10A also includes a network interface (NWI) 16 thatcouples processing node 10A to a wired, wireless or hybrid network,which may be a standardized network such as Ethernet, or a proprietarynetwork or interconnect bus. Other processing nodes 10B-10D are ofidentical construction in the exemplary embodiment, but embodiments ofthe invention may be practiced in asymmetric distributed systems havingnodes with differing features. Although only four compute nodes 10A-10Dare illustrated, a distributed computer system in accordance with anembodiment of the present invention will generally include a largenumber of compute nodes connected via one or more networks. Thedistributed computer system of FIG. 1 also includes other resources suchas I/O devices 19, including graphical display devices, printers,scanners, keyboards, mice, which may be coupled to the network or one ofnodes 10A-10D via workstation computers that provide a user interface toadministrative personnel and other users. Nodes 10A-10D are also coupledto storage devices 18, for storing and retrieving data and programinstructions, such as storing computer program products in accordancewith an embodiment of the invention.

Referring now to FIG. 2, communication between multiple nodes 20 of thedistributed computer system of FIG. 1 is shown. As mentioned above,nodes 20 may correspond exactly on a one-to-one basis with processingnodes 10A-10D (and other nodes) of FIG. 1, or nodes 20 may bepartitioned in a different manner as virtual processing nodes. Forexample, a single node 20 may have exclusive use of multiple processingnodes, e.g. nodes 10A-10B, and result in a system having a greaternumber of virtual nodes than processing nodes, or alternatively,multiple nodes 20 may be implemented on a single processing node, e.g.,node 10A. In the present invention, each of nodes 20 represents at leastone operating system image and one or more applications executing withinthe operating system image. In general, the entire system as depictedmay execute a single application, but sub-tasks within the applicationare apportioned to the various nodes 20, which may be identicalsub-tasks or different sub-tasks. The present invention concerns nodestatus monitoring and communications of node status among nodes withinthe computer system of FIG. 1. Gossip messages and other node statuscommand messages are communicated between nodes 20 and the messages areprocessed at each node. Operational messages are also passed betweennodes 20, including data and program code transmissions. Nodes aregenerally arranged in a cluster, which is a group of virtual or physicalprocessing nodes organized to perform a particular task or group oftasks, e.g., for a particular customer. As illustrated, communicationbetween nodes in a cluster may be accomplished by direct node to nodecommunications 22 or by next-neighbor communications 24 in which nodes20 pass along messages to other nodes 20. While the description of theparticular embodiments of the invention below describes the node statusand gossip communications taking place node-to-node as opposed to fromnodes to a centralized status facility, it is understood thatnode-to-node communications do not require that the transmissions takeplace over a physical network that does not traverse other nodes, andthe next-neighbor communications 24 can be used to implementnode-to-node communications. Further, as alternative to the illustratedcommunications is to use a central facility for inter-nodecommunication, which is distinct from centrally reporting and acting onnode status indications. Therefore, the present invention may encompasstechniques that use a centralized communication point, such as a stararrangement, while still passing gossip messages in a node-to-nodemanner with respect to the messaging endpoints. However, since theillustrated embodiment provides a de-centralized communicationsmonitoring system, it is generally preferable not to use a centralexchange for messaging.

Referring now to FIGS. 3A-3C, a node status methodology in accordancewith an embodiment of the invention is illustrated. In the illustratedembodiment, four special messages are used to perform the statusmonitoring:

CMD_GOSSIP{ <node, seq#>, <node, seq#> ...<node, seq#>, } CMD_NODE_DOWN{node } CMD_ALIVE{ node } CMD_RXT{node, seq#}Gossip message CMD_GOSSIP is transmitted between all of the node pairson a regular basis within the cluster of nodes. For each node that hasan operational status at the transmitting node, the node number and thesequence number of the most recently received reliable, i.e.,operational, transmission are inserted in the message data. The gossipmessage itself is not considered a reliable transmission, as the gossipmessages form part of the status monitoring, so the sequence numberrepresents the last actual operational transmission from the specifiednode. When a node receives a gossip message from another node, the nodenumbers and sequence numbers provided from the other node are used toupdate a local database, which may be a table, that contains the statusand message sequence number information for the other nodes, as well asan indication of the time elapsed since the node has received a gossipmessage from each other node. There are essentially two checks in thisscheme: first, the nodes expect to receive, and time the reception of,gossip messages from each other node in the cluster; and second, thenodes receive information about all nodes in the cluster and their mostrecently received operational messages from each other node in thecluster, providing a very robust status sharing methodology. FIG. 3Aillustrates transmission of gossip messages CMD_GOSSIP from nodes 20Cand 20D to the other nodes. Nodes 20A and 20B also transmit gossipmessages, but those have been omitted from the illustration for clarity.In FIG. 3A node 20D is shown as not transmitting a gossip message tonode 20B, for example due to loss of a suitable route, or a hardwarefailure. The lack of transmission of a gossip message to node 20B fromnode 20D will be used to illustrate the other messages and techniques ofthe present invention with reference to FIGS. 3B-3C below.

Node status message CMD_NODE_DOWN is transmitted from a node to allother nodes in the cluster when a node determines that a gossip messagehas not been received from a particular node for a predetermined timeperiod. The time period may be determined from a timer, a comparison ofa time count to a stored timestamp corresponding to the particular node,a count of gossip message intervals determined from gossip messagesreceived from other nodes (other than the particular node), or othersuitable technique for determining that too much time has passed sincethe last gossip message from the particular node in question. In FIG.3B, node 20B eventually determines, due to the lack of transmissions ofgossip messages from node 20D as illustrated in FIG. 3A, and albeiterroneously, that node 20B is non-operational. Node 20B then transmitsnode status messages CMD_NODE_DOWN containing the node number of node20B, to nodes 20A and 20C. When nodes 20A and 20C receive the nodestatus messages CMD_NODE_DOWN from node 20B, nodes 20A and 20C transmitnode status message CMD_ALIVE containing the node number of node 20B,which causes node 20D to correct the node status for node 20B and maytrigger additional action such as forming a new route between node 20Band 20D to re-establish communications. If a node actually is down(i.e., the other nodes do not receive gossip messages at all from thatnode), then the nodes stop including the node marked as down in thegossip messages.

Message CMD_RXT is used to obtain operational messages from another nodethat were somehow missed. Each node retains operational messages inretransmit queues, one for each other node in the cluster. The entriesin the retransmit queues are only removed when their sequence numbershave been indicated in gossip messages as having been received by eachnode in the cluster. Since each gossip message contains sequence numbersfor each node, each gossip message causes the latest sequence number foreach node to be updated. If the sequence number changes for a node, thecorresponding retransmit queue is inspected and entries are removed upto and including the message indicated by the newly entered sequencenumber. Thus, message CMD_RXT and the data retention in queues at eachnode provide an acknowledgement system that ensures that at leasttemporary communications errors are recoverable. FIG. 3C illustratesnode 20A receiving a gossip message that node B (corresponding to node20B) has received operational message with sequence number 5. Node 20Ahas not received the message with sequence number 5 only the messagewith sequence number 4, and so node 20A sends a retransmit messageCMD_RXT to node 20B with sequence number 5. Node 20B responds with acopy of message with sequence number 5 from the one of retransmit queues22 that corresponds to node 20A.

Referring now to FIG. 4, a node status monitoring method in accordancewith an embodiment of the present invention is illustrated. A nodereceives a message from another node (step 50). If the message is agossip message (decision 51), the sequence numbers for the nodes areupdated in the local database/table and the timer for the originatingnode is reset (step 52). If a sequence number for a node is greater thanthe last reliable message received from the node (decision 53) then aretransmit request is sent to that node (step 54), in order to obtainthe missed reliable transmission. If the message is instead a node downmessage (decision 55), if the timer has elapsed for the specified node(decision 56) then the status of the node is marked as down and the nodeis removed from further gossip messages (step 57). Otherwise, if thetimer has not elapsed for the specified node (decision 56), then a nodealive message is sent to the other nodes for the specified node (step58). If the message is instead a node alive message (decision 59), thenthe status of the specified node is marked as up (operational) and thenode resumes including the specified node in gossip messages (step 60).If the message is a retransmit message (decision 61), then the queueentry for the originating node is re-transmitted (step 62). Otherwise,if the message is a reliable (operational) message it is processed (step63). The illustrated method is the information flow for messageprocessing. Not illustrated is the handling of elapsing gossip messagetimers, which can be performed synchronously with message processing orasynchronously. If the timer elapses for a node, its local status ismarked as down and a node status down message is transmitted.

As noted above, the present invention may be embodied as a system,method, and/or a computer program product. A computer program productmay be embodied in firmware, an image in system memory or anothermemory/cache, stored on a fixed or re-writable media such as an opticaldisc having computer-readable code stored thereon. Any combination ofone or more computer readable medium(s) may be used to store the programinstructions in accordance with an embodiment of the invention. Thecomputer readable medium may be a computer readable signal medium or acomputer readable storage medium. A computer readable storage medium maybe, for example, but not limited to, an electronic, magnetic, optical,electromagnetic, infrared, or semiconductor system, apparatus, ordevice, or any suitable combination of the foregoing. More specificexamples (a non-exhaustive list) of the computer readable storage mediumwould include the following: an electrical connection having one or morewires, a portable computer diskette, a hard disk, a random access memory(RAM), a read-only memory (ROM), an erasable programmable read-onlymemory (EPROM or Flash memory), an optical fiber, a portable compactdisc read-only memory (CD-ROM), an optical storage device, a magneticstorage device, or any suitable combination of the foregoing.

In the context of the present application, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device. A computer readable signal medium may include apropagated data signal with computer readable program code embodiedtherein, for example, in baseband or as part of a carrier wave. Such apropagated signal may take any of a variety of forms, including, but notlimited to, electro-magnetic, optical, or any suitable combinationthereof. A computer readable signal medium may be any computer readablemedium that is not a computer readable storage medium and that cancommunicate, propagate, or transport a program for use by or inconnection with an instruction execution system, apparatus, or device.Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, or any suitable combination of theforegoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages.

While the invention has been particularly shown and described withreference to the preferred embodiments thereof, it will be understood bythose skilled in the art that the foregoing and other changes in formand details may be made therein without departing from the spirit andscope of the invention.

1. A method for determining node operating status among a cluster ofnodes of a computer system, the method comprising: transmitting gossipmessages directly between node pairs in the cluster of nodes, whereinthe gossip messages contain an indication of operational status of othernodes in the cluster of nodes, wherein the other nodes are nodes otherthan the nodes in the node pairs; receiving the gossip messages at thenode pairs; responsive to the receiving, at the nodes, updatingoperating status of other nodes according to the received gossipmessages, wherein the status of a particular one of the other nodes isset to a non-operating status if the receiving has not received a gossipmessage from the particular node during a predetermined time period; andresponsive to setting the status of the particular one of the othernodes to a non-operating status, transmitting a node down messageindicating the non-operating status of the particular node to the othernodes in the cluster.
 2. The method of claim 1, further comprising: at afirst node other than the particular node, receiving the node downmessage; responsive to receiving the node down message, determiningwhether or not the first node has received a gossip message from theparticular node during the predetermined time period; and responsive todetermining that the first node has received the gossip message from theparticular node during the predetermined time period, transmitting anode alive message from the first node indicating that the particularnode is operating.
 3. The method of claim 2, wherein the node alivemessage is transmitted to multiple nodes in the cluster.
 4. The methodof claim 2, further comprising, at a second node, receiving the nodealive message from the first node, wherein the updating, responsive tothe receiving the node alive message from the first node, sets thestatus of the particular node at the second node to an operating status.5. The method of claim 1, wherein the gossip messages contain sequencenumbers of the most recent transmission received from the other nodes,and wherein the updating updates the sequence numbers in a localstorage.
 6. The method of claim 5, further comprising: responsive toreceiving at a first node a particular gossip message containing a firstsequence number associated with a second node, determining whether thefirst sequence number is greater than a second sequence number of themost recently received message from the second node; and responsive todetermining that the first sequence number is greater than the secondsequence number, transmitting a retransmit command to the second node.7. The method of claim 1, wherein the transmitting transmits gossipmessages containing indications of operational status for nodes havingan operational status set as operating at the nodes originating thegossip messages.